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1.0  Introduction 


The  Joint  Advanced  Distributed  Simulation  Joint  Test  and  Evaluation  (JT&E)  was  chartered  by 
the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and  Evaluation)1,  Office  of 
the  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in  October  1994  to  investigate  the 
utility  of  advanced  distributed  simulation2 3  (ADS)  technologies  for  support  of  developmental  test 
and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  The  program  is  Air  Force 
lead  with  Army  and  Navy  participation  and  is  scheduled  to  end  in  March  2000. 

The  JADS  JT&E  charter  focuses  on  three  issues:  what  is  the  present  utility  of  ADS,  including 
distributed  interactive  simulation  (DIS),  for  test  and  evaluation  (T&E);  what  are  the  critical 
constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E;  and  what  are  the 
requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a  more  complete 
T&E  capability  in  the  future. 

The  JADS  JT&E  investigated  ADS  applications  in  three  slices  of  the  T&E  spectrum:  the  System 
Integration  Test  (SIT)  explored  ADS  support  to  air-to-air  missile  testing;  the  End  to  End  (ETE) 
Test  investigated  ADS  support  to  command,  control,  communications,  computers,  intelligence, 
surveillance  and  reconnaissance  (C4ISR)  testing;  and  the  Electronic  Warfare  (EW)  Test  examined 
ADS  support  for  EW  testing.  The  JADS  Joint  Test  Force  (JTF)  was  also  chartered  to  observe  or 
participate  at  a  modest  level  in  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an 
effort  to  broaden  conclusions  developed  in  the  three  dedicated  tests. 

A  key  finding  of  the  JADS  JT&E  was  that  the  primary  challenges  to  developing  and  executing  a 
distributed  test  are  programmatic  rather  than  technical.  The  requirement  to  interact  with  multiple 
facilities  and  organizations  with  their  associated  processes  presents  potential  problems  over  the 
full  range  of  test  planning  from  concept  development  to  implementation.  This  special  report  is  a 
companion  report  to  the  JADS  report,  A  Test  Planning  Methodology  —  From  Concept 
Development  Through  Test  Execution.  The  report  mirrors  the  ADS-based  test  planning  and 
implementation  steps  and  provides  insight  into  the  challenges  a  program  or  test  manager  will 
encounter  during  each  step.  The  report  is  formatted  with  extracts  from  A  Test  Planning 
Methodology  —  From  Concept  Development  Through  Test  Execution  identified  in  italics. 


1  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation,  Strategic  and  Tactical  Systems. 

2  ADS  is  a  networking  method  that  permits  the  linking  of  constructive  simulations  (digital  computer  models), 
virtual  simulations  (man-in-the-loop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed 
locations  into  a  single  environment/scenario.  Such  linking  can  result  in  a  more  realistic,  safer,  and/or  more 
detailed  evaluation  of  the  system  under  test. 

3  ADS  is  a  networking  method  that  permits  the  linking  of  constructive  simulations  (digital  computer  models), 
virtual  simulations  (man-in-the-loop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed 
locations  into  a  single  environment/scenario.  Such  linking  can  result  in  a  more  realistic,  safer,  and/or  more 
detailed  evaluation  of  the  system  under  test. 
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2.0  General  Challenges  for  the  Program  Manager 

A  program  manager  considering  the  use  of  distributed  simulation  or  distributed  testing  to 
overcome  test  limitations  will  be  faced  with  two  general  challenges  throughout  the  test  planning 
and  implementation  process.  The  first  is  cultural  bias  within  the  acquisition  and  test  and 
evaluation  communities;  the  second  is  the  lack  of  experience  with  distributed  testing  technology 
within  the  communities.  The  program  manager  must  be  aware  of  these  challenges  and  be 
prepared  to  mitigate  the  impact  of  these  on  test  planning. 

Cultural  bias  manifests  itself  in  several  ways  to  impact  the  decisions  made  in  the  test  planning  and 
implementation  process.  While  these  biases  may  not  be  explicitly  stated,  they  will  often  be  the 
basis  behind  the  input  and  advice  the  program  manager  receives  from  the  test  and  evaluation 
community. 

•  Traditional  test  methodologies  are  adequate.  Processes  and  methodologies  for  testing  specific 
types  of  systems  have  been  developed  and  institutionalized  over  decades  and  through  the 
testing  of  multiple  systems.  These  processes  tend  to  be  sequential  in  nature  and  based  on  the 
maturity  of  the  system  under  test  and  the  capabilities  of  the  ranges/facilities.  The  range/facility 
engineers  and  the  system/test  engineers  have  accepted  these  processes  and  their  inherent 
limitations  for  so  long  that  they  have  problems  conceiving  of  different  ways  to  test  a  system. 
The  challenge  will  be  to  get  the  test  and  evaluation  community  to  accept  the  need  to  develop 
new  processes  and  technologies  to  overcome  test  limitations. 

•  All  testing  must  be  conducted  using  native  spectrum.  This  is  a  bias  particular  to  those  systems 
that  interact  with  both  friendly  or  threat  systems  using  radio  frequency,  infrared,  or  other 
spectra.  Since  the  technologies  for  distributing  testing  or  distributed  simulation  typically  link 
facilities  using  digital  transmission,  these  interactions  will  have  to  be  converted  into  a  digital 
format  prior  to  transmission.  The  challenge  will  be  to  convert  the  native  spectrum  in  such  a 
way  as  to  avoid  unacceptable  degradation  of  signal  and  to  gain  acceptance  from  the  traditional 
test  community  that  the  interactions  remain  valid. 

•  We  can  meet  your  test  requirements  at  our  facility/range.  The  test  and  evaluation 
infrastructure  of  the  Department  of  Defense  (DoD)  has  been  developed  over  decades  by  the 
services  to  test  specific  systems.  Since  the  mid-1980s,  the  defense  drawdown  and  budget  cuts 
have  resulted  in  considerable  reduction  of  facilities  and  ranges,  primarily  to  reduce  perceived 
duplication  of  capability  between  the  services.  As  a  result,  test  facilities  and  ranges  are 
reluctant  to  admit  any  limitations  in  their  capabilities  to  meet  test  requirements.  Additionally, 
the  competition  among  ranges/facilities  for  business  has  resulted  in  thinly  veiled  animosity 
among  the  range/facilities  and  only  minimal  cooperation.  Finally,  the  services,  to  protect  their 
remaining  test  resources,  have  established  policies  and  processes  that  make  it  difficult  for  an 
acquisition  program  to  conduct  testing  at  another  service’s  range  or  facility.  All  of  these 
combine  to  work  against  a  program  manager  attempting  to  overcome  test  limitations  through 
the  linking  of  test  facilities. 
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Distributed  testing  and  distributed  simulation  have  actually  been  used  within  the  test  and 
evaluation  communities  for  many  years.  All  system  integration  laboratories,  hardware-in-the-loop 
facilities,  installed  systems  test  facilities,  and  even  open  air  ranges  conduct  distributed  testing  as  a 
normal  process  and  usually  incorporate  a  level  of  distributed  simulation  within  their  test 
environments.  Some  of  these  facilities  have  established  elaborate  networks  to  link  various 
laboratories  and  facilities  at  a  single  installation  into  combined  test  environments.  However,  very 
few  of  these  facilities  have  accepted  that  the  methodologies  they  use  within  their  facilities  are 
distributed  testing  and  distributed  simulation  and  can  be  extended  to  link  to  other  facilities.  This 
lack  of  acceptance  may  be  due  to  a  lack  of  experience  at  the  facilities  and  ranges  and  a  tradition  of 
bottom-up  planning  of  tests  based  on  the  specific  capabilities  of  a  specific  laboratory,  test  facility, 
or  range.  The  development  of  a  distributed  test  environment  requires  the  combination  of  system 
under  test,  facility,  range,  network,  instrumentation,  and  analysis  functions  lead  by  a  strong 
systems  engineering  function  focused  on  providing  the  best  test  environment  for  the  system  under 
test  (SUT).  This  process  is  inherently  a  top-down  process  driven  by  the  operational  requirements 
of  the  system  rather  than  existing  test  capabilities.  JADS  has  seen  little  evidence  of  test 
organizations  establishing  processes  that  will  support  planning  and  implementation  of  distributed 
test  environments.  The  program  manager  will  have  to  take  the  lead  to  develop  the  experience 
base  to  navigate  through  the  planning  and  implementation  of  a  distributed  test. 
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3.0  ADS-Inclusive  Test  Concept  Development  Methodology 

The  methodology  described  in  the  JADS  report,  A  Test  Planning  Methodology  —  From  Concept 
Development  Through  Test  Execution,  used  an  example  which  was  couched  in  terms  of  OT&E. 
But,  as  pointed  out  in  the  report,  as  OT&E  moves  left  on  the  acquisition  timeline  and  as  new 
systems  demand  ever  more  complex  test  environments,  the  process  is  applicable  to  DT&E  as  well. 

The  methodology  makes  an  assumption  that  ADS  can  technically  support  representation  of  the 
military  operating  environment  at  the  campaign  or  theater  level.  If  ADS  is  included  in  the  test 
planning  tool  kit  from  the  outset,  it  is  possible  to  begin  the  test  concept  development  process  at 
the  top  rather  than  the  bottom.  (The  "top  ”  may  not  be  at  theater  level;  it  is  established  by  the 
relevant  operational  task  or  tasks.)  The  methodology  described  in  this  paper  is  a  top-level 
methodology.  It  is  an  approach  which  is  compatible  with  the  "strategy  to  task”  or  “mission- 
level  evaluation  ”  philosophy.  It  is  also  a  methodology  for  test  concept  development  which 
incorporates  the  consideration  of  ADS  —  it  is  not  an  ADS  planning  methodology.  This 
methodology  is  designed  to  provide  insights  on  whether  to  use  ADS  and  where  in  a  test  program 
the  use  might  fit. 

The  advantage  of  a  top-down  approach  to  test  concept  development  is  that  the  whole  gamut  of 
interactions  is  available  for  consideration  even  if  many  of  those  interactions  are  assessed  as 
irrelevant  and  excluded  from  the  final  concept.4  The  top-down  approach  doesn ’t  require  that 
every  possible  interaction  be  included  in  the  test,  but  it  does  require  an  item  by  item  assessment 
of  each  interaction.  Decisions  to  exclude  interactions  are  conscious  decisions  not  default 
decisions  as  a  function  of  a  bottom-up  approach. 

Mission-  or  task-level  evaluation  is  explicitly  a  top-down  approach.  The  top  level,  for  test 
planning  purposes,  may  be  much  lower  than  campaign  or  theater.  Just  how  high  the  top  level  is, 
is  a  function  of  the  task  being  evaluated.  Some  systems  may  have  little  or  no  interaction  beyond 
a  unit  boundary,  and  others  may  interact  closely  with  the  theater  and  campaign  levels.  In  the 
case  of  DT&E,  it  is  necessary  to  substitute  “ specification  sets”  for  "tasks.  ”  The  substitution 
should  not  be  difficult.  While  there  may  be  evolutionary  changes  as  a  program  evolves,  the 
operational  tasks  expected  of  a  new  system  are  known  as  a  result  of  mission  needs  analysis  and 
serve  as  the  basis  for  initial  requirements  development.  It  shouldn ’t  be  hard  to  map  certain 
system  specifications  to  a  specific  task.  The  methodology,  as  described,  should  be  useful  for 
most  DT&E. 

Given  that  the  methodology  applies  to  both  DT&E  and  OT&E,  and  that  the  trend  in  T&E  is  to 
consolidate  DT&E  and  OT&E  whenever  possible,  it  follows  that  the  most  appropriate  application 
of  the  methodology  is  to  an  entire  program,  from  concept  exploration  to  production,  deployment, 
and  operations  support.  Although  the  methodology  could  be  applied  to  a  single  acquisition  phase 


4  A  top-down  approach  will  not  help  if  it  is  implemented  with  a  mind  set  fixed  on  historical  limitations.  It’s 
necessary  for  the  test  planners  to  understand  that  ADS  provides  opportunities  which  weren’t  previously  feasible. 
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or  even  to  a  single  test,  this  paper  will  focus  on  the  development  of  a  test  plan  that  spans  the  life 
of  a  program. 

The  logic  flow  for  the  initial  elements  of  the  concept  development  methodology  is  shown  in 
Figure  1. 
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Although  the  test  concept  development  process  is  presented  in  a  sequential  format  and  flow,  the 
actual  implementation  of  the  process  will  likely  be  conducted  in  a  parallel  fashion.  The  degree  of 
parallelism  will  be  driven  by  the  available  resources  and  the  experience  level  of  the  concept 
development  team.  The  program/test  manager  will  have  to  carefully  control  the  process  to  insure 
each  task  is  carefully  examined  and  the  appropriate  test  concept  developed. 

3.1  Step  1.  Understanding  the  System  Under  Test 

Step  1  requires  the  test  planners  to  research  the  acquisition  documentation  to  gain  a  thorough 
understanding  of  the  SUT  and  its  intended  operating  environment.  This  understanding 
incorporates  the  operational  tasks  the  system  is  designed  to  perform,  the  critical  system 
parameters,  the  system  and  operational  requirements,  the  concept  of  operations,  the  logistical 
support  concept,  and  the  top  level  or  general  operating  environment.  One  piece  of  the 
understanding  deals  with  the  technical  or  specification  aspects  of  the  system.  The  other  piece 
deals  with  the  interactions  between  the  technical  characteristics  of  the  system  and  the  world  it 
operates  in  from  a  strategic  perspective  —  the  friendly  and  supporting  forces,  the  natural 
environment,  and  the  threats  posed  by  the  enemy. 

The  task  of  understanding  the  SUT  presents  the  first  challenge  to  the  program  or  test  manager. 
The  level  of  documentation  available  during  the  concept  exploration  or  program  definition  and 
risk  reduction  phases  of  a  new  program  is  typically  limited  to  a  mission  needs  statement  and 
possibly  a  draft  of  the  operational  requirements  document.  These  documents  may  not  adequately 
describe  the  operational  tasks,  the  concept  of  operations,  or  the  operating  environment.  The 
program  manager  will  need  the  support  of  the  requiring  operational  commands,  both  within  the 
service(s)  and  within  the  unified  command  structure,  to  help  define  the  operating  environment  and 
the  interactions  expected  of  the  system.  Additionally,  since  it  is  likely  the.  operating  environment 
will  include  other  new  or  upgraded  systems,  the  program  manager  will  be  required  to  interact  with 
other  programs  to  understand  the  capabilities  and  interactions  required  by  these  systems. 

3.2  Step  2.  Select  a  Task 

This  step  involves  the  selection  of  a  specific  task.  A  complex  system  may  be  assigned  many 
operational  tasks.  Some  tasks  may  be  very  similar,  while  others  may  be  vastly  different.  It  is 
possible  that  similar  tasks  may  be  grouped  for  evaluation  purposes  and  tested  on  the  basis  of  a 
single  task. 

Even  though  the  program  and  test  manager  may  have  carefully  worked  through  Step  1  and 
believe  they  understand  the  system  under  test,  they  cannot  proceed  through  the  planning  process 
in  a  vacuum.  The  planning  process  must  be  conducted  with  both  developmental  and  operational 
testers  and  will  probably  require  support  from  the  requiring  commands.  The  manager  must 
carefully  coordinate  and  control  this  process  to  ensure  a  top-down  process  uninhibited  by 
historical  limitations,  cultural  bias,  or  hidden  agendas. 
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3.3  Step  3.  Develop  Relevant  Measures 

Once  a  specific  task  is  selected,  the  planners  can  develop  relevant  measures  for  the  task  and  a 
task-specific  operating  environment.  The  operating  environment  in  combination  with  assigned 
objectives  and  missions  provides  a  context  for  the  test  measures  and  defines  the  cast  of  players. 
In  order  to  structure  a  test,  the  player  cast  has  to  be  embedded  in  a  dynamic  operational 
scenario.  The  scenario  supports  detailed  mission  layout  activities  and  time-sequenced  events 
for  the  SUT.  The  scenario  developed  in  Step  3  is  an  operational  scenario;  a  real  world  scenario 
—  not  a  test  scenario. 

In  this  step,  the  challenge  to  the  manager  is  to  control  his  planning  team  and  to  produce  a  real 
world  operational  scenarios.  Experienced  test  managers  and  engineers  involved  in  the  planning 
will  be  aware  of  the  potential  limitations  of  the  various  ranges  and  facilities.  The  temptation  will 
be  to  narrow  the  scenario  toward  the  known  test  capabilities.  A  potential  solution  to  this 
challenge  is  to  allow  only  the  operational  representatives  to  actively  participate  in  this  step.  This 
would  allow  the  testers  to  better  understand  the  operational  scenario  while  avoiding  potential 
narrowing  of  the  scenario. 

3.4  Step  4.  Consider  Using  ADS 

The  activity  described  to  this  point  is  simply  a  test  planning  or  test  concept  development 
approach.  Step  4  is  a  switch  point  —  to  include  or  not  to  include  a  detailed  consideration  of 
ADS  use  as  part  of  the  concept  development  methodology. 

At  this  time  the  test  planners  will  move  from  the  real-world  operational  scenario  to  a  series  of  test 
scenarios.  Each  test  scenario  will  have  certain  assumptions  concerning  the  test  objectives 
appropriate  for  a  particular  phase  of  the  development.  For  example,  in  the  program  definition  and 
risk  reduction  phase,  the  planners  will  develop  scenarios  to  examine  the  trade  space  for  the 
program.  Later,  in  the  engineering  and  manufacturing  development  phase  a  similar  test  scenario 
will  be  used  to  evaluate  the  actual  performance  of  components  and  eventually  the  SUT.  For  each 
phase  of  testing,  the  program  manager  must  lead  the  T&E  representatives  to  determine  if  an 
adequate  test  environment  can  be  represented  using  traditional  test  resources.  If  the  task  can  be 
adequately  and  affordably  represented  and  the  measures  evaluated,  then  traditional  resources  and 
methodology  may  be  the  preferred  approach.  However,  if  the  test  planners  are  reasonably  certain 
the  test  environment  cannot  be  adequately  represented  using  the  traditional  test  approaches,  then 
they  have  two  choices:  accept  the  test  limitations  or  explore  ADS  to  see  if  the  technology  can 
make  a  better  test  within  the  fiscal  constraints.  This  phase  may  require  representatives  of  the 
appropriate  test  ranges  and  facilities  to  join  the  test  planning  team. 

Stated  succinctly,  the  decision  associated  with  Step  4  is  about  the  adequacy  of  the  test  scenario 
as  compared  with  the  operational  scenario.  In  a  world  with  no  fiscal  or  safety  constraints,  the 
operational  scenario  and  the  test  scenario  would  be  identical.  In  the  real  world,  the  issue 
becomes  “can  we  approximate  reality  with  sufficient  accuracy  to  have  a  satisfactory  test.  If  the 
test  planners  cannot  provide  an  appropriate  test  environment  using  traditional  test  approaches, 
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then  the  answer  is  “no, "  and  the  planners  should  explore  whether  ADS  can  do  anything  to 
improve  the  situation.  If  the  answer  is  “no,  ”  then  the  process  moves  along  to  step  5.  If  the 
answer  is  “yes,  we  can  provide  a  suitable  test  environment,  ”  then  the  planners  can  proceed  with 
a  traditional  test  plan  for  that.  Particular  operational  task.  Other  tasks  may  require  different 
approaches. 

3.5  Step  5.  Select  a  Player 

Traditional  testing  shortfalls  often  include  an  insufficient  number  of  test  articles,  insufficient 
number  of  threats,  and  inadequate  representation  of  friendly  force  interactions.  The  process  of 
ADS  exploration  begins  with  a  visit  to  the  player  list  developed  in  Step  3,  and  the  first  player  on 
that  list  is  the  SUT.  Depending  on  where  the  program  is,  the  SUT  may  be  available  in  a  variety 
of  forms.  Early  in  the  program,  the  SUT  may  only  be  available  as  a  digital  system  model 
(DSM).  Later  the  SUT  may  exist  in  brassboard  form  with  a  variety  of  subcomponents  scattered 
among  a  variety  of  vendors.  Eventually  the  SUT  will  be  available  in  prototype  or  production 
version  form,  and  a  training  simulator  version  will  emerge.  (The  DSM  version  is  still  available 
at  this  stage.) 

At  each  stage  of  the  program,  the  challenge  for  the  program  manager  and  test  planners  is  to  select 
the  appropriate  representation  of  the  SUT.  This  selection  will  normally  be  biased  toward  the  most 
current,  highest  fidelity  representation,  however,  a  lower  fidelity  version  could  be  selected  based 
on  the  test  objectives  required  to  test  a  particular  operational  task.  Early  in  a  program,  the  DSM 
version  of  the  SUT  will  exhibit  the  performance  identified  in  the  specifications  or  may  have  the 
ability  to  support  trade-offs  of  performance  values.  Since  the  performance  of  a  DSM  SUT  is 
fixed,  this  version  is  not  appropriate  for  evaluating  actual  performance.  However,  such  a 
representation  is  appropriate  for  answering  questions  concerning  the  impact  a  given  level  of 
performance  has  on  the  operational  scenario.  Later  in  the  program,  performance  measurements 
will  be  made  using  brassboard  prototypes,  but  it  may  be  more  cost  effective  to  update  the  DSM  to 
measured  performance  and  use  it  for  some  operational  evaluations.  Similar  decisions  will  be  made 
for  each  representation  of  the  SUT  developed  during  the  life  of  the  program.  For  some  test 
scenarios,  two  different  representations  of  the  SUT  having  the  same  performance  characteristics, 
e.g.,  a  DSM  and  a  brassboard,  may  be  used  to  represent  a  certain  concepts  of  operations.  The 
selection  of  SUT  representation  with  its  associated  fidelity  for  each  test  scenario  is  the  driver  for 
the  next  step  in  the  process  —  selecting  other  players. 

3.6  Step  6.  Determine  Player  Representation 

The  determination  of  a  player  representation  will  be  made  based  on  both  the  availability  of 
representations  and  the  test  objectives.  This  step  addresses  both  of  these  criteria.  The  planning 
team  will  need  to  investigate  the  available  representations  for  each  player  beginning  with  the 
SUT.  The  planning  team  will  also  have  to  define  the  requirements  for  the  test  to  meet  the  test 
objective.  After  looking  at  the  SUT  configuration  choices,  the  planners  can  move  on  to  the  other 
players.  If  the  player  list  is  prioritized  on  the  basis  of  the  more  direct  interactions,  then  the  more 
important  players  are  addressed  first.  For  each  player,  the  test  planners  must  have  access  to 
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information  about  the  various  manifestations  of  the  player.  They  must  know  what  forms  are 
available,  and  they  must  learn  what  they  can  about  capabilities  and  costs  for  each  form. 

Step  6  involves  a  lot  of  research  and  learning. 

Representations  of  the  major  players  will  not  be  limited  to  the  various  test  ranges  and  facilities. 
The  program  manager  and  planners  will  have  to  investigate  a  wide  variety  of  facilities  to  find  the 
appropriate  representations.  Research  laboratories  and  battle  laboratories  represent  two  classes 
of  facilities  that  provide  a  rich  source  of  capabilities  with  varying  fidelity  for  the  test  planner.  The 
modeling  and  simulation  agencies  are  also  sources  for  information  concerning  various 
representations.  Both  the  Defense  Modeling  and  Simulation  Organization  (DMSO)  and  the 
service  agencies  have  developed  modeling  and  simulation  resource  repositories  (MSRR)  to  store 
and  maintain  information  about  various  representations  and  about  previous  environments  using 
these  capabilities.  Another  source  to  investigate  is  the  training  community  for  the  particular 
player.  Finally,  the  opportunity  may  exist  to  connect  with  five  players.  The  emphasis  of  the 
research  should  be  to  find  all  the  representations  rather  than  settling  for  what  the  test  ranges  and 
facilities  have  available. 

The  program  manager  will  not  only  have  to  find  the  various  representations,  but  will  also  have  to 
determine  what  capabilities  and  fidelity  each  representation  can  bring  to  his  event.  Considering 
that  most  facilities  operate  on  some  sort  of  fee  for  service  arrangement,  it  is  often  a  challenge  for 
the  program  manager  to  get  by  the  marketing  and  gain  complete  understanding  of  a  particular 
facility’s  actual  capabilities  and  fidelity. 

The  level  of  research  involved  in  this  step  may  require  the  program  manager  to  implement  a  level 
of  parallelism  in  the  process.  While  research  is  being  conducted,  the  existence  of  a  player  can  be 
assumed  and  the  team  can  skip  to  Step  8,  continuing  the  iteration  until  an  initially  adequate 
environment  is  developed.  Additionally,  other  tasks  can  be  examined  in  parallel  with  the  initial 
outputs  being  fed  into  the  research  process  in  Step  6.  Finally,  the  team  will  return  to  completing 
Steps  6  through  8  for  each  player  and  each  task. 

3.7  Step  7.  Fidelity  Versus  Cost 

Step  7  involves  the  art  of  compromise  between  fidelity  and  cost.  With  the  information  gleaned  in 
Step  6,  the  test  planners  are  in  a  position  to  make  reasoned  choices  about  the  players  in  the  test 
and  the  appropriate  form  of  representation  for  each  of  them.  Rough  order  of  magnitude  (ROM) 
costs  are  adequate  in  the  test  concept  development  process.  Costs  are  refined  in  detailed  test 
planning. 

The  program  manager  is  limited  in  this  step  by  the  ability  to  get  accurate  information  in  Step  6. 
The  challenge  for  this  step  is  to  accept  the  limitations  and  attempt  to  identify  the  level  of  fidelity 
required  and  the  levels  of  fidelity  available.  The  purpose  is  to  develop  a  test  concept.  During 
detailed  planning,  all  these  initial  compromises  will  be  revisited  and  better  information  may  lead  to 
different  answers.  A  subtle  point  to  remember  is  that  fidelity  and  cost  are  not  always  directly 


10 


related.  For  example,  a  hardwareVman-in-the-loop  laboratory  may  prove  to  be  more  expensive  to 
operate  than  a  live  system  depending  on  how  the  live  system  is  connected. 

3.8  Step  8.  Evaluate  Adequacy  of  the  Environment 

When  the  process  reaches  Step  8,  the  question  facing  the  planners  involves  the  adequacy  of  the 
environment  which  will  be  created  by  the  interactions  of  the  players  selected.  Each  time  Steps  5 
through  7  are  executed,  the  planners  must  ask  whether  more  players  are  needed.  If  so,  they 
return  to  Step  5.  Step  5  is  executed  repeatedly  until  the  test  planners  are  satisfied  that  the  test 
environment  is  rich  enough  in  terms  of  meaningful  interactions  to  support  a  sound  test. 

Once  the  initial  test  environment  is  defined,  the  test  planners  should  review  the  fidelity  choices 
and  evaluate  the  interactions  between  each  of  the  players.  The  iteration  of  steps  5  through  7  will 
define  a  set  of  players  based  on  the  fidelity  required  from  each  player  by  the  SUT.  However, 
most  operational  tasks  will  also  require  some  or  all  the  players  to  interface  in  some  form  with  one 
another.  The  challenge  is  to  ensure  that  all  the  interactions  among  all  the  players  are  meaningful 
with  respect  to  the  objectives  of  the  test. 

3.9  Step  9.  Initial  Planning 

When  the  test  planners  are  satisfied  with  the  test  concept  for  a  given  operational  task,  they  can 
proceed  with  initial  test  planning  for  the  associated  test. 

Initial  test  planning  is  conducted  to  the  level  necessary  to  support  the  requirements  of  the  master 
test  plan  to  be  developed  in  Step  11.  For  an  acquisition  program,  the  master  test  plan  will  be  the 
test  and  evaluation  master  plan  developed  according  to  the  requirements  in  DoD  Regulation 
5000.2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition  Programs  (MDAPS)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs,  Appendix  III. 

3.10  Step  10.  Additional  Tasks 

Step  10  involves  the  examination  of  the  functionalities  of  the  SUT,  and  the  assessment  of  the 
necessity  for  further  testing  for  additional  operational  tasks.  If  there  are  additional  operational 
tasks  that  differ  enough  from  those  already  addressed  in  test  planning  to  this  point,  then  the 
planners  need  to  loop  back  to  Step  2  of  this  process  and  develop  another  test  concept.  Planning 
cannot  move  on  to  Step  11  until  initial  planning  has  been  completed  for  all  operational  tasks. 

Step  10  can  be  performed  sequentially  after  each  iteration  of  Step  9  is  completed,  or  if  there  are 
enough  planning  resources,  planning  for  additional  tasks  can  go  on  in  parallel.  In  any  case,  test 
planning  is  not  complete  until  all  the  relevant  operational  tasks  have  been  addressed. 
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3.11  Step  11.  Develop  Master  Plan 

The  task  associated  with  Step  11  involves  the  deconfliction  and  coordination  of  each  of  the 
individual  task-oriented  test  segments.  Essentially  Step  11  involves  the  development  of  the 
master  plan  and  schedule. 

The  most  challenging  aspect  of  this  step  is  the  coordination  of  individual  task-oriented  test 
segments.  The  test  planners  need  to  develop  a  process  to  compare  the  test  segments  against  one 
another  to  determine  where  a  single  environment  may  be  suitable  for  evaluating  multiple  tasks. 
This  comparison  may  involve  adding  players  to  one  or  more  environments  to  establish  the 
superset  required  for  multiple  tasks. 

Since  the  master  plan  covers  the  entire  life  of  the  program,  another  aspect  of  coordination  and 
scheduling  that  should  be  considered  is  the  evolutionary  aspects  of  the  environment(s).  The 
program  manager  should  plan  to  evolve  simple  environments  used  early  in  a  program  into  the 
more  complex  environments  used  for  later  test  events.  Just  as  the  program  manager  develops  a 
risk  management  approach  for  the  development  of  the  system,  so  should  a  risk  management 
approach  for  the  development  of  the  test  environment  be  developed.  Finally,  the  program 
manager  should  consider  the  ability  to  transition  portions  of  the  test  environment  to  the  training 
system. 


12 


4.0  ADS-Based  Test  Planning  and  Implementation  Methodology 


Assuming  the  decision  is  made  to  implement  ADS-based  testing,  the  following  methodology 
applies.  This  methodology  follows  the  steps  given  in  the  high  level  architecture  (HLA) 
federation  development  and  execution  process  (FEDEP)  model  [Ref  1J.  In  comparing  these 
guidelines  with  the  FEDEP  model,  note  that  the  terms  “ ADS  architecture’’  and  “distributed 
test”  used  here  equate  to  the  term  “federation”  in  the  FEDEP  model,  and  the  terms  “facilities,  ” 
“participants,  ”  and  “players  ”  used  here  equate  to  the  term  “federates  ’’  in  the  FEDEP  model. 

The  FEDEP  model  groups  the  activities  needed  to  develop  and  execute  a  distributed  test  into  six 
steps. 

•  Step  1:  The  test  sponsor  or  evaluator  and  the  distributed  test  development  team  define  and 
agree  on  a  set  of  objectives  and  document  what  must  be  accomplished  to  achieve  those 
objectives.  This  is  a  test  planning  step  and  is  addressed  by  the  test  planning  methodology. 

•  Step  2:  A  representation  of  the  real  world  domain  of  interest  is  developed  and  described  in 
terms  of  a  set  of  required  objects  and  interactions.  Most  of  the  activities  under  this  step  are 
addressed  by  the  test  planning  methodology. 

•  Step  3:  Distributed  test  participants  (federates)  are  determined,  and  required  functionalities 
are  allocated  to  the  participants. 

•  Step  4:  The  federation  object  model  (FOM)  is  developed  (if  HLA  is  implemented), 
participant  agreements  on  consistent  databases/algorithms  are  established,  and 
modifications  to  federates  are  implemented  (as  required). 

•  Step  5:  All  necessary  distributed  test  implementation  activities  are  performed,  and  testing  is 
conducted  to  ensure  interoperability  requirements  are  being  met. 

•  Step  6:  The  distributed  test  is  executed,  outputs  are  generated,  and  results  provided. 

The  FEDEP  model  breaks  the  six  steps  into  activities,  as  shown  in  Table  1. 
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Table  1.  Distributed  Test  Planning  and  Implementation  Activities 


STEP 

ACTIVITIES 

1.  Define  Distributed  Test  Objectives 

1.1  Identify  Needs 

1.2  Develop  Objectives 

2.  Develop  Conceptual  Model 

2.1  Develop  Scenario 

2.2  Perform  Conceptual  Analysis 

2.3  Develop  Test  Requirements 

3.  Design  Distributed  Test 

3. 1  Select  Participants 

3.2  Allocate  Functionality 

3.3  Prepare  Plan 

4.  Develop  Distributed  Test 

4.1  Develop  FOM 

4.2  Establish  Participant  Agreements 

4.3  Implement  Participant  Modifications 

5.  Integrate  and  Test  Architecture 

5.1  Plan  Execution 

5.2  Integrate  and  Test  ADS  Architecture 

6.  Execute  Distributed  Test  and  Analyze 
Results 

6.1  Execute  Distributed  Test 

6.2  Process  Output 

6.3  Prepare  Results 

Although  the  FEDEP  and  the  distributed  test  planning  and  execution  guidelines  presented  in  this 
report  and  its  companion  report  are  presented  in  sequential  step  format,  the  program/test  manager 
must  accept  that  the  actual  process  is  a  system  engineering  process  with  the  associated  iteration 
between  steps.  Additionally,  JADS’  experience  demonstrates  a  considerable  amount  of 
parallelism  during  steps  1  through  3  and  again  during  steps  3  through  5.  JADS  recommends  the 
development  of  a  detailed  project  schedule  to  manage  these  activities. 

4.1  Step  1.  Define  Distributed  Test  Objectives 

According  to  the  FEDEP  model,  the  purpose  of  the  activities  for  this  step  is  to  define  and 
document  a  set  of  needs  that  are  to  be  addressed  through  the  development  and  execution  of  a 
distributed  test  and  to  transform  these  needs  into  a  more  detailed  list  of  specific  test  objectives. 
The  key  activities  for  this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  2. 


Figure  2.  Define  Distributed  Test  Objectives 


At  the  beginning  of  this  process,  the  program  manager  needs  to  evolve  the  test  concept  planners 
into  the  integrated  product  team  (IPT)  that  will  conduct  the  detailed  planning  and  implementation 
of  the  distributed  test.  Every  organization  directly  involved  in  the  distributed  test  should  be 
represented  on  the  team.  The  members  on  the  team  must  have  the  authority  to  speak  for  their 
organizations  and  must  have  the  ability  to  bring  the  appropriate  experts  from  their  organizations 
to  provide  input  to  the  IPT.  JADS’  experience  identifies  two  key  members  of  the  IPT.  The 
chairman  of  the  IPT  must  have  the  authority  to  make  all  decisions  relating  to  the  test  content, 
schedule,  and  resources.  The  chairman’s  primary  assistant  is  the  system  integrator,  the  lead 
technical  person  responsible  for  the  integration  of  the  various  organizations/facilities  involved. 
Given  the  overall  concept  is  a  distributed  test  environment  that  spans  multiple  phases  of  a 
program,  the  program  manager  must  exercise  great  care  in  selecting  the  IPT  leadership  and 
membership  and  nurturing  the  team  throughout  its  existence. 

The  program  manager  should  also  consider  the  various  uses  of  the  environment  over  the  life  of  a 
system  and  ensure  the  IPT  has  representatives  from  each  of  the  user  groups.  For  example,  if  the 
environment  will  be  used  for  requirements  development  and  engineering  trade  studies  early  in  the 
program,  the  IPT  will  require  representatives  from  the  operational  and  engineering  communities  in 
addition  to  the  test  community.  Likewise,  if  there  is  a  potential  to  evolve  the  parts  or  all  the 
distributed  test  environment  into  a  distributed  training  environment,  the  training  community  will 
also  need  to  be  on  the  IPT. 

4.1.1  Activity  1.1  -  Identify  Needs 

According  to  the  FEDEP  model,  the  primary  purpose  of  this  activity  is  to  develop  a  clear 
understanding  of  the  problem  to  be  addressed  by  the  distributed  test.  Inputs  to  this  activity  are 
the  program  objectives  and  information  on  resources  available  to  support  a  distributed  test.  The 
main  output  of  this  activity  is  a  needs  statement  which  includes  the  following. 

•  High-level  descriptions  of  critical  systems  of  interest 

•  Coarse  indications  of fidelity  and  required  behaviors  for  simulated  players 

•  Key  events  that  must  be  represented  in  the  distributed  test  scenario 

•  Output  data  requirements 

•  Resources  that  will  be  available  to  support  the  distributed  test  (e.g.,  funding,  personnel, 
tools,  facilities) 

•  Any  known  constraints  which  may  affect  how  the  distributed  test  is  developed  (e.g.,  due 
dates,  security  requirements) 

Although  the  test  concept  development  phase  examined  each  of  these  areas  in  some  level  of 
detail,  the  IPT  should  carefully  review  and  validate  the  test  concept  prior  to  proceeding  into 
detailed  planning.  Since  the  IPT  has  been  expanded  to  include  all  the  affected  organizations,  this 
will  ensure  the  assumptions  made  during  test  concept  development  are  valid  and  confirm  the 
information  used  to  make  decisions. 
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4.1.2  Activity  1.2  -  Develop  Objectives 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  refine  the  needs  statement  into 
a  more  detailed  set  of  specific  objectives  for  the  distributed  test.  This  activity  requires  close 
collaboration  between  the  distributed  test  user/sponsor  and  the  test  development  team  to  ensure 
that  the  resulting  objectives  meet  the  stated  needs.  The  user/sponsor  must  clearly  define, 
communicate,  and  document  test  requirements  early  in  the  test  planning  phase.  The  main  input 
to  this  activity  is  the  needs  statement  from  the  previous  activity.  The  main  outputs  of  this  activity 
are  a  statement  of  the  test  objectives  and  initial  planning  documents.  The  test  objectives 
statement  should  include  the  following  information. 

•  A  prioritized  list  of  measurable  test  objectives 

•  A  high-level  description  of  key  ADS  architecture  characteristics  (e.g.,  repeatability, 
portability,  time  management  approach) 

•  Needed  equipment,  facilities,  and  data 

•  Operational  context  constraints  or  preferences,  including  friendly/threat/civilian  order  of 
battle,  geographic  regions,  environmental  conditions,  and  tactics 

•  Identification  of  security  position,  including  estimated  security  level  and  possible  designated 
approval  authority 

•  A  configuration  management  approach 

•  Identification  of  tools  to  support  scenario  development,  conceptual  analysis,  verification, 
validation  and  accreditation  ( W&A )  and  test  activities,  and  configuration  management 

4.2  Step  2.  Develop  Conceptual  Model 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  develop  an  appropriate 
representation  of  the  real-world  domain  that  applies  to  the  distributed  test  environment  and  to 
develop  the  test  scenario.  During  this  step,  test  objectives  are  transformed  into  a  set  of  specific 
requirements  for  use  as  success  criteria  during  ADS  architecture  testing.  The  key  activities  for 
this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  3. 
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Figure  3.  Develop  Conceptual  Model 
4.2.1  Activity  2.1  -  Develop  Scenario 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  a  functional 
specification  of  the  test  scenario.  The  primary  input  to  this  activity  is  the  operational  context 
constraints  specified  in  the  test  objectives  statement,  although  existing  scenario  databases  may 
also  provide  a  reusable  starting  point  for  scenario  development.  The  primary  output  is  the  test 
scenario.  The  scenario  description  should  include  the  following. 

•  The  types  and  numbers  of  major  players  that  must  be  represented  in  the  distributed  test 

•  A  functional  description  of  the  capabilities,  behavior,  and  relationships  among  these  major 
players  over  time 

•  A  specification  of  relevant  environmental  conditions  that  impact  or  are  impacted  by  players 
in  the  distributed  test 

•  Initial/terminal  conditions  and  the  specific  map  projection  chosen  for  the  scenario 

Note  that  most  of  these  items  should  have  been  determined/developed  during  application  of  the 
test  concept  development  methodology.  However,  their  determination/development  should  be 
repeated  here  to  check/validate  the  earlier  findings. 

It  is  critically  important  that  the  “scenario”  be  approved  by  an  appropriate  authority.  Ideally,  the 
selected  scenario  will  have  been  previously  developed  and  approved  for  other  purposes,  e.g., 
developing  defense  planning  guidance.  However,  even  with  an  approved  scenario,  the  program 
manager  will  have  to  ensure  it  meets  the  requirements  for  the  test.  For  example,  the  scenario  used 
in  the  JADS  End-to-End  Test  had  previous  approval  by  the  U.S.  Army  Training  and  Doctrine 
Command  for  use  in  Army  OT&E.  When  JADS  evaluated  the  scenario  for  use  with  Joint 
Surveillance  Target  Attack  Radar  System  (Joint  STARS),  we  determined  the  order  of  battle  and 
movements  failed  to  model  rear  area  activities,  e.g.,  logistics,  at  the  appropriate  level  of  fidelity 


for  our  test.  JADS  worked  with  the  U.S.  Army  Test  and  Experimentation  Command  to  expand 
the  scenario  to  include  appropriate  rear  area  movements  and  actions.  Additionally,  the  expanded 
scenario  was  then  run  through  another  model  to  produce  the  appropriate  intelligence  reports  for 
the  movements.  Not  only  was  the  resulting  scenario  used  for  the  JADS  test,  but  it  is  also  being 
used  for  follow-on  testing  within  both  the  Army  and  Air  Force. 

4.2.2  Activity  2.2  -  Perform  Conceptual  Analysis 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  produce  a  conceptual  model  of 
the  ADS  environment.  The  primary  inputs  to  this  activity  are  the  test  scenario  from  the  previous 
activity,  the  test  objectives  statement,  and  any  doctrine  and  tactics  appropriate  for  the  scenario. 
The  output  of  this  activity  is  the  conceptual  model  which  provides  an  implementation- 
independent  representation  that  serves  as  a  vehicle  for  transforming  objectives  into  functional 
and  behavioral  capabilities,  and  provides  a  crucial  traceability  link  between  the  test  objectives 
and  the  design  implementation.  The  conceptual  model  is  a  description  of  the  players,  their 
actions,  and  any  interactions  among  players  that  need  to  be  included  in  the  distributed  test  in 
order  to  achieve  all  test  objectives.  These  are  described  without  any  reference  to  specific 
simulations  that  will  be  used. 

The  primary  challenge  to  the  IPT  for  this  activity  is  to  develop  a  conceptual  model  which  is 
actually  implementation  independent.  As  discussed  previously,  the  IPT  is  made  up  of 
experienced  testers  and  of  representatives  from  the  appropriate  ranges  and  facilities.  Each  of 
these  brings  with  them  a  database  of  capabilities  which  is  likely  to  impact  the  conceptual  model. 
At  this  stage,  the  conceptual  model  should  not  be  limited  by  the  capabilities  of  a  specific  range, 
facility,  or  simulation.  It  should  accurately  reflect  the  test  objectives  and  appropriate  doctrine  and 
tactics. 

4.2.3  Activity  2.3  -  Develop  Test  Requirements 

According  to  the  FEDEP  model,  the  conceptual  model  will  lead  to  the  definition  of  detailed 
distributed  test  requirements  and  test  evaluation  criteria.  These  requirements  should  be  based 
on  the  distributed  test  objectives,  should  be  directly  testable,  and  should  provide  the 
implementation-level  guidance  needed  to  design  and  develop  the  distributed  test  (cost  impact 
factor  of  rank  #9  -  see  Appendix  A).  The  test  requirements  will  also  be  the  basis  for  the  criteria 
for  evaluating  test  results  (see  Fig.  3).  Major  top-level  requirements  which  should  be  addressed 
include  the  following  (although  some  of  these  requirements  should  have  been  developed  during 
application  of  the  test  planning  methodology,  their  development  should  be  repeated  here  to 
check/validate  the  earlier  findings). 

•  Fidelity  requirements 

-  The  fidelity  requirements  for  all  players  represented  in  the  distributed  test  scenarios  must 
be  determined.  The  required  fidelity  depends  upon  the  maturity  of  the  SUT,  the  SUT  test 
objectives,  and  the  nature  of  the  interactions  between  the  SUT  and  the  other  players. 

-  The  fidelity  of  the  SUT  representation  may  be  limited  to  available  models  or  test  articles. 
For  example,  during  early  developmental  test  and  evaluation  (DT&E '),  a  low-fidelity 
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digital  model  may  be  the  only  SUT  representation  available,  but  during  late  DT&E  and 
OT&E,  possible  SUT  representations  may  include  high-fidelity  digital  models,  hardware- 
in-the-loop  (HWIL)  labs,  and  live  test  articles.  If  multiple  SUT  representations  are 
available  with  varying  levels  of  fidelity,  the  choice  will  usually  be  driven  by  the  SUT  test 
objectives  and  other  considerations  such  as  availability  and  cost. 

-  The  required  fidelity  for  the  other  players  normally  depends  on  the  fidelity  of  the  SUT, 
the  sensitivity  of  test  objectives/measures  to  player  interactions  with  the  SUT,  the  strength 
of  the  interactions  with  the  SUT  (players  that  have  strong,  or  tightly  coupled,  interactions 
with  the  SUT  will  generally  have  higher  fidelity  requirements  than  those  which  do  not), 
the  test  objectives,  and  cost  and  availability  considerations. 

-  It  is  important  to  involve  SUT  experts  from  the  beginning  of  the  distributed  test  program 
in  order  to  determine  fidelity  requirements,  establish  the  data  and  instrumentation 
requirements,  verify/validate  the  analytical  approach,  assist  in  the  development  of  test 
matrices  and  test  procedures,  and  provide  overall  SUT  expertise.  The  support  of  more 
than  one  SUT  expert  should  be  planned  for  (and  budgeted  for)  throughout  the  test. 

•  Interaction  requirements 

-  Use  the  conceptual  model  to  determine  the  data  types  that  must  be  exchanged  among 
players  to  permit  interaction  including  entity  state  data,  tactical  messages,  launch  and 
detonation  indications  (if  appropriate),  and  trial  start  and  stop  notification. 

•  Latency  requirements  (cost  impact  factor  of  rank  #4  -  see  Appendix  A) 

-  Determine  the  maximum  acceptable  latency  and  latency  variations  for  each  pair  of 
interacting  players.  The  maximum  latency  requirement  will  be  determined  by  how  closely 
coupled  the  interactions  are  and  by  the  maximum  allowable  error  in  the  location  of  one 
player  as  perceived  by  the  other. 

The  determination  of  a  latency  budget  can  be  one  of  the  more  challenging  engineering  problems 
for  the  IPT.  The  tendency  for  most  engineers  is  to  underestimate  the  amount  of  allowable  latency 
for  a  particular  problem.  JADS’  experience  is  that  the  latency  budget  can  usually  be  larger  than 
originally  estimated.  Many  factors  drive  the  estimate  and  the  IPT  must  carefully  consider  every 
part  of  the  problem.  The  program  manager  must  ensure  the  IPT  carefully  develops  these 
requirements. 

•  Data  reliability  requirements 

-  Determine  the  maximum  acceptable  level  of  ADS-induced  errors,  such  as  dropout  rate 
and  out-of-order  data  messages.  The  allowable  errors  may  vary  with  data  types.  For 
example,  some  loss  of  entity  state  data  may  be  tolerable  for  short  durations  if  dead 
reckoning  can  supply  the  missing  data  within  acceptable  error  tolerances.  However,  the 
loss  of  a  single  discrete  message  may  invalidate  an  entire  trial.  This  determination  may 
drive  the  reliability  requirement  for  data  transport  and  guide  the  selection  of 
data 

transport  protocols. 
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•  Data  analysis  requirements 

-  Draft  a  preliminary  data  management  and  analysis  plan  (DMAP)  that  details  the 
analysis  approach  for  each  test  objective.  From  the  DMAP  determine  which  data  must 
be  collected  and  the  analysis  techniques  to  be  applied. 

One  of  the  key  findings  from  the  JADS’  experiences  is  that  data  collection  and  analysis  in  a 
distributed  environment  can  be  very  different  from  conventional  test  methods.  Distributed  testing 
will  typically  produce  considerably  more  data  per  unit  of  time  than  live  testing.  Add  this  to  the 
availability  of  more  interacting  units  per  segment  of  time  and  the  data  collection  or  data  analysis 
process  can  be  overcome  with  data.  Finally,  consider  that  laboratories  typically  have  more 
opportunities  to  instrument  systems  than  when  the  systems  are  operated  in  a  live  mode.  This 
potential  problem  needs  to  be  considered  when  evaluating  analysis  techniques  and  tools. 

After  all  these  requirements  have  been  developed,  the  capability  of  the  support  agencies  (e.g., 
simulation  or  range  facilities,  networking  and  engineering  team)  to  support  the  test  must  be 
clearly  stated  and  documented,  such  as  by  a  statement  of  capability  (SOC).  The  SOC  documents 
the  set  of  requirements  and  provides  a  clear  statement  of  the  support  agency’s  capabilities, 
constraints,  and  limitations  in  meeting  those  requirements. 

A  key  challenge  for  the  LPT  during  this  phase  will  be  to  provide  requirements  to  the  supporting 
organizations  in  a  consistent  format.  The  Major  Range  and  Test  Facility  Base  has  developed  a 
universal  standard  that  allows  for  submission  of  the  requirements  to  the  range/facility  in  the  form 
of  a  program  introduction  document  (PID).  Unfortunately,  other  training  and  laboratory  facilities 
have  not  adopted  this  standard  and  the  use  of  the  standard  throughout  the  test  community  is  not 
consistent.  JADS  recommends  that  the  IPT  adopts  a  standard  practice  which  will  be  used  for  all 
the  facilities  and  organizations.  The  PID  format  should  be  adequate  for  the  IPT  to  document  the 
requirements  for  each  of  the  facilities  in  the  distributed  environment. 

The  SOC  mentioned  above  is  the  corresponding  document  to  the  PID.  In  this  case,  the 
supporting  organization  describes  its  understanding  of  the  requirements  and  its  capabilities  to 
meet  the  requirements.  If  upgrades  or  modifications  are  required  to  the  facility,  the  SOC  should 
provide  a  detailed  understanding  of  the  scope  of  the  changes  and  the  resources  and  schedule 
required.  For  facilities  not  familiar  with  these  documents,  the  program  manager  may  be  required 
to  lead  the  organization  into  providing  adequate  information. 

The  support  agencies  also  need  to  create  an  integrated,  detailed  work  breakdown  structure 
(WBS)  early  in  the  program  which  is  consistent  with  the  SOC.  It  is  also  important  to  have 
accurate  cost  estimates  allocated  against  the  WBS  tasks  in  order  to  help  make  program 
management  decisions. 

This  requirement  will  place  a  challenge  on  the  supporting  facilities.  Many  of  the  available 
facilities  are  not  familiar  with  developing  work  breakdown  structures  to  be  used  to  estimate  the 
resources  required  and  to  track  the  development  of  the  environment.  Another  problem  will  be  the 
ability  to  provide  an  estimate  with  the  appropriate  level  of  fidelity.  For  example,  the  normal 
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practice  for  a  particular  test  facility  is  to  estimate  costs  based  on  a  week  of  testing.  This  is  based 
on  years  of  experience  at  the  facility.  However,  when  such  a  facility  is  integrated  into  a 
distributed  environment,  it  may  be  more  appropriate  to  estimate  costs  based  on  hourly  or  daily 
usage,  especially  during  integration  testing. 

4.3  Step  3.  Design  Distributed  Test 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  identify,  evaluate,  and  select  all 
distributed  test  participants  (federates),  allocate  required  functionality  to  those  participants,  and 
develop  a  detailed  plan  for  test  bed  development  and  implementation.  The  key  activities  for  this 
step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  4. 


Test  Requirements 


4.3.1  Activity  3.1  -  Select  Participants 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  determine  the  suitability  of 
individual  player  representations  (e.g.,  simulations,  HWIL  labs,  or  live  players/ranges)  to 
become  participants  in  the  distributed  test.  The  input  to  this  activity  is  the  conceptual  model 
developed  in  Activity  2.2.  The  output  is  an  identification  of  the  specific  player  representations 
selected. 

This  activity  involves  the  identification  of  specific  simulations,  HWIL  labs,  or  live  players/ranges 
to  be  used  in  the  distributed  test  and  their  locations.  This  selection  is  driven  primarily  by  the 
following  factors. 
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•  Perceived  ability  of  potential  representations  to  represent  the  players’  behavior  and  the 
interactions  specified  in  the  conceptual  model 

•  Fidelity  requirements  for  each  player 

•  Managerial  constraints,  such  as  availability  cost,  schedule,  and  security  considerations 

•  Technical  constraints,  such  as  W&A  status  and  portability 

•  For  live  players,  the  selection  of  particular  test  ranges  is  also  driven  by  considerations  of 
range  instrumentation  quality  and  quantity  and  data  processing  capability 

The  primary  challenge  in  this  step  relates  to  truth  in  advertising.  All  test  facilities  and  most  other 
facilities  owe  their  continued  existence  to  their  ability  to  attract  customers.  As  a  result,  die 
program  manager  will  have  to  carefully  question  each  candidate  facility  to  determine  both  the 
capabilities  and  limitations  of  the  facility.  At  the  same  time  it  is  also  useful  to  look  at  other  factors 
which  may  impact  your  test  event.  These  factors  include  previous  experience  with  linking, 
previous  accreditations,  instrumentation,  time  synchronization  capability  and  data  collection  and 
analysis  capabilities. 

4.3.2  Activity  3.2  -  Allocate  Functionality 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  allocate  the  responsibility  to 
represent  the  entities  and  actions  in  the  conceptual  model  to  the  participants.  This  activity  will 
allow  for  the  assessment  of  whether  the  set  of  selected  participants  provides  the  full  set  of 
required  functionality  or  whether  one  or  more  of  the  representations  will  need  to  be  enhanced  to 
meet  the  distributed  test  requirements.  The  inputs  to  this  activity  are  the  identified  participants 
from  the  previous  activity,  along  with  the  test  requirements,  the  test  scenario,  and  the  conceptual 
model.  The  output  is  allocated  requirements  for  the  participants,  including  any  requirements  for 
modifying  existing  player  representations  or  designing  new  ones. 

Once  requirements  are  allocated  and  carefully  and  consistently  documented,  the  program/test 
manager  can  enter  into  formal  discussions  with  the  supporting  facilities  and  organizations.  It  is  at 
this  step  when  the  PID  is  provided  to  the  supporting  facility,  and  the  facility  then  produces  a  SOC 
with  the  associated  estimates  for  necessary  modifications.  While  adding  formality  to  the  process 
will  likely  require  additional  schedule,  it  is  important  for  the  program/test  manager  to  understand 
early  in  the  process  the  scope  of  the  modifications  with  the  associated  risk. 

Another  issue  that  must  be  addressed  by  the  IPT  is  the  tasking  of  operational  units  or  assets. 
During  the  allocation  of  requirements,  some  of  the  players  will  be  live  and  the  program/test 
manager  will  have  to  arrange  for  this  support.  Each  of  the  services  has  different  processes  by 
which  operational  resources  can  be  tasked  in  support  of  a  test  event. 

The  U.S.  Army  uses  the  operational  test  plan  (OTP)  and  the  test  requirements  council  (TRC)  to 
identify,  prioritize,  and  task  operational  units  in  support  of  T&E.  Although  this  process  provides 
a  well-defined  and  structured  approach,  the  program  manager  and  IPT  must  be  careful  to  identify 
all  the  required  support,  including  support  for  test  environment  integration  and  testing  prior  to  the 
actual  test  event(s).  Another  challenge  for  the  program  manager  is  to  maintain  continued 
coordination  of  test  requirements  and  details  with  the  operational  unit.  Tasking  to  support  a 
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distributed  test  will  often  be  viewed  as  an  additional,  unfunded  requirement  on  the  tasked  unit. 
The  program  manager  can  ensure  the  best  support  by  continually  striving  to  make  participation  in 
the  test  a  positive  experience  for  the  tasked  unit. 

The  other  services  have  less  structured  processes  for  identifying,  prioritizing,  and  tasking 
operational  units  to  support  testing  of  a  system.  Although  requirements  for  supporting  tests  are 
documented  in  various  program  documents,  e.g.,  program  management  directive  or  test  and 
evaluation  master  plan,  the  program  manager  or  the  operational  test  manager  is  primarily 
responsible  to  establish  agreements  with  major  commands  for  the  support  of  a  specific  test  event. 
The  negotiation  and  continued  care  and  feeding  of  these  agreements  can  present  a  time- 
consuming  challenge  to  the  program  manager. 

4.3.3  Activity  3.3  -  Prepare  Plan 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  a  coordinated  plan  to 
guide  the  development,  test,  and  execution  of  the  distributed  test.  The  inputs  to  this  activity  are 
the  initial  planning  documents  prepared  during  the  development  of  the  test  objectives  (Activity 
1.2)  and  the  allocated  participant  requirements.  The  output  is  the  detailed  planning  documents. 

An  old  adage  says,  “The  job  is  not  over  until  the  paperwork  is  finished.”  Likewise  the  planning  is 
not  over  until  the  detailed  plans  are  written.  Good  planning  is  a  task  that  takes  considerable  time 
and  effort.  The  program/test  manager’s  challenge  is  to  find  the  time  and  resources  to  devote  to 
writing  the  detailed  plans.  Numerous  reasons  will  be  found  to  avoid  this  task:  “We  don’t  know 
enough  yet.”  “That  is  part  of  our  normal  process.”  We  can  streamline  the  process  if  we  do  not 
have  to  develop  this  plan.”  Regardless,  the  process  of  writing  the  plan  will  allow  the  managers  to 
identify  what  they  do  not  know,  what  really  is  a  normal  process,  and  what  resources  and  effort  are 
really  required  to  complete  the  test. 

4.4  Step  4.  Develop  Distributed  Test 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  develop  the  FOM  (if  HLA  is  to  be 
implemented),  modify  the  simulations/range  facilities  if  necessary,  and  prepare  the  distributed 
architecture  for  integration  and  test.  The  key  activities  for  this  step  and  the  activity  inputs  and 
outputs  are  shown  in  Figure  5. 
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Conceptual  Model 


According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  the  FOM.  The  inputs 
to  this  activity  are  the  detailed  planning  documents  and  the  allocated  participant  requirements. 
The  outputs  are  the  FOM  and  federation  execution  data  (FED)  file,  if  appropriate. 

4.4.2  Activity  4.2  -  Establish  Participant  Agreements 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  establish  all  agreements  among 
participants  necessary  for  a  fully  consistent,  interoperable,  distributed  simulation  environment. 
The  inputs  to  this  activity  are  the  test  scenario,  the  conceptual  model,  and  the  FOM  (if  HLA  is  to 
be  implemented).  The  output  is  revised  participant  allocated  requirements,  including  any 
requirements  for  additional  modifications. 

During  previous  steps  the  IPT  has  developed  detailed  plans,  completed  the  PID/SOC  process,  and 
developed  an  interface  control  document  (ICD).  Each  of  these  has  increased  the  understanding  of 
how  the  various  facilities  and  organizations  will  operate  with  one  another.  During  this  step,  the 
team  must  arrive  at  final  agreements  and  issue  the  appropriate  documentation  to  begin  the  actual 
development  of  the  environment.  Each  of  the  plans  should  be  coordinated  and  agreed  upon  by  all 
the  parties.  Each  organization  must  agree  to  operate  in  accordance  with  the  ICD.  Finally,  the 
program/test  manager  should  complete  the  actual  work  agreements  with  each  of  the  supporting 
facilities  and  organizations.  For  facilities  that  use  the  PID/SOC  process,  this  may  be  as  simple  as 
agreeing  to  the  SOC  and  issuing  funding.  For  other  organizations,  a  memorandum  of  agreement 
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or  understanding  may  be  required,  either  as  a  cover  document  to  the  PID/SOC  or  as  a  complete 
replacement. 

During  this  activity,  the  program/test  manager  may  also  run  into  problems  related  to  type  or  color 
of  money.  Acquisition  programs  are  usually  funded  with  research,  development,  test  and 
evaluation  (RDT&E)  appropriations.  As  a  result,  test  facilities  and  organizations  are  organized  to 
use  these  types  of  appropriations  to  fund  their  operations.  However,  the  representations  of 
players  in  the  test  may  be  resident  at  laboratories,  training  facilities,  and  operational  units  and 
some  of  these  organizations  may  require  operation  and  maintenance  (O&M)  funding  rather  than 
RDT&E.  The  program/test  manager  needs  to  ascertain  what  color  of  money  is  required  and  work 
with  the  financial  organization  to  ensure  the  appropriate  funding  is  available. 

4.4.3  Activity  4.3  -  Implement  Participant  Modifications 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  implement  participant 
modifications  identified  in  previous  activities.  The  input  to  this  activity  is  the  updated  allocated 
participant  requirements.  The  output  is  the  modified  participants. 

4.5  Step  5.  Integrate  and  Test  Architecture 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  plan  the  test  execution,  establish 
all  required  interconnectivity  between  the  nodes/players,  and  test  the  network  prior  to  execution. 
The  key  activities  for  this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  6. 


Modified  Participants 


Figure  6.  Integrate  and  Test  Architecture 
4.5.1  Activity  5.1  -  Plan  Execution 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  define  and  develop  the  full  set 
of  information  required  to  support  the  distributed  test  execution.  The  inputs  to  this  activity  are 
the  FOM  and  federation  execution  data  (FED)  file,  if  appropriate,  the  test  scenario,  and  the 
detailed  planning  documents.  The  outputs  are  a  refined  and  detailed  integration  test  plan, 
W&A  plan,  and  test  procedures. 
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Once  all  the  agreements  are  signed  and  all  the  development  schedules  are  known,  the  EPT  can 
begin  the  next  level  of  planning.  The  integration  plan  documents  a  logical  build-up  of  capabilities 
based  on  needs  and  development  schedules.  Each  integration  event  needs  to  be  carefully  planned 
to  accomplish  key  tasks  and  reduce  risk  for  the  actual  tests.  Additionally,  this  activity  brings  in 
another  group  that  will  almost  certainly  provide  additional  challenges  for  the  program/test 
manager. 

The  normal  security  paradigm  for  certification  and  accreditation  is  focused  at  the  facility  level.  A 
person  or  organization  within  each  facility  is  responsible  for  documenting  and  maintaining  the 
security  configuration  of  the  facility.  This  person  certifies  the  configuration  to  the  designated 
approval  authority  (DAA)  for  accreditation  of  the  facility  to  operate  in  that  configuration.  The 
problems  arise  when  facilities  are  linked  together.  Now  the  accreditation  authority  for  the  linked 
environment  is  shared  between  the  DAAs  of  each  of  the  facilities.  The  challenge  for  the 
program/test  manager  is  to  get  each  of  the  DAAs  to  agree  that  the  environment  provides  adequate 
protection  for  the  data  and  facilities.  JADS  developed  memoranda  of  agreement  among  the 
various  DAAs  to  obtain  this  agreement.  More  recently,  DMSO  sponsored  the  development  of  a 
security  overlay  to  the  FEDEP.  This  overlay  is  based  on  the  Department  of  Defense  Information 
Technology  Security  Certification  and  Accreditation  Process  (DITSCAP)  and  is  applied  to  the 
FEDEP.  The  overlay  outlines  the  steps  required  during  each  FEDEP  activity  which  will  build  up 
to  a  joint  accreditation  of  the  environment. 

4.5.2  Activity  5.2  -  Integrate  and  Test  ADS  Architecture 

This  activity  combines  the  separate  FEDEP  activities  of  ’‘integrate  federation  ”  and  “test 
federation”  because  of  the  close  connection  between  the  two.  An  iterative  “test-fix-test” 
approach  is  recommended,  so  that  the  integration  and  test  activities  become  closely  interrelated. 
According  to  the  FEDEP  model,  the  purpose  of  these  activities  is  to  bring  all  the  distributed  test 
participants  into  a  unifying  operating  environment  and  to  test  that  they  can  all  interoperate  to 
the  degree  required  to  achieve  the  test  objectives.  The  inputs  to  this  activity  are  the  detailed 
integration  test  plan,  W&A  plan,  and  test  procedures.  The  outputs  are  refined  test  procedures 
(to  be  used  during  test  execution),  W&A  results,  and  an  ADS  architecture  that  has  been 
thoroughly  tested  and  is  ready  for  test  execution.  , 

During  the  integration  and  test  activity  the  program/test  manager  faces  three  related  challenges. 
The  first  is  the  challenge  of  scheduling  adequate  time  for  integration  and  test.  We  make  the  basic 
assumption  that  most  of  the  facilities  used  in  the  test  environment  are  not  controlled  by  the 
program/test  manager.  We  also  make  the  assumption  that  the  facilities  operate  under  a  high 
percentage  of  utilization.  Given  these  assumptions,  for  each  integration  and  test  event  the 
program/test  manager  must  work  with  at  least  two  facilities  to  coordinate  a  schedule  for  the 
event.  This  can  be  very  challenging.  For  example,  integration  for  Phase  3  of  the  JADS  EW  Test 
needed  to  be  conducted  between  December  1998  and  April  1999.  JADS  required  several  one-  or 
two-day  periods  for  integration  and  testing  between  JADS  and  the  Air  Combat  Environment  Test 
and  Evaluation  Facility  (ACETEF).  Additionally,  immediately  prior  to  the  actual  test,  JADS 
required  an  additional  two  to  three  days  of  full  dress  rehearsal  with  all  three  facilities  on  line. 
During  the  same  period,  all  the  facilities  were  supporting  other  tests.  Scheduling  these  activities 
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took  a  lot  of  coordination  among  the  facility  managers  to  ensure  the  integration  and  test  activities 
were  conducted  at  the  appropriate  times.  A  positive  aspect  of  this  problem  is  that,  with  the 
exception  of  the  full  dress  rehearsals,  most  integration  and  test  activities  do  not  require  complete 
configurations  and  may  be  able  to  be  conducted  in  parallel  with  other  activities  at  the  facility. 

Network  scheduling  and  prioritization  may  also  present  a  challenge  to  the  program/test  manager. 
Current  DoD  policy  requires  networking  to  be  conducted  over  Defense  Information  Systems 
Network  (DISN)  common-user  networks  wherever  possible.  One  of  these  services,  the  Defense 
Simulation  Internet  (DSI),  has  a  scheduling  process  that  allows  the  user  to  schedule  periods  of 
usage  where  a  certain  level  of  service  is  provided  to  the  user.  This  essentially  adds  another 
schedule  which  must  be  coordinated.  Other  services  are  available  at  all  times  but  may  not  be  able 
to  support  the  required  levels  of  service.  In  some  cases,  e.g.,  Secret  Internet  Protocol  Router 
Network  (SIPRNET),  a  user  basically  takes  the  chance  that  performance  will  be  adequate  for  the 
test  at  a  scheduled  time.  Other  networks  provide  scaleable  services  which  adjust  based  on  traffic. 
The  program/test  manager  must  be  aware  of  the  type  of  service  that  will  be  provided  and  allow 
for  any  potential  impacts  in  planning.  In  light  of  these  considerations  and  the  lessor  capabilities 
offered  when  JADS  conducted  its  planning,  the  JADS  approach  was  to  lease  dedicated 
commercial  T-l  communications  lines. 

Another  challenge  for  the  program/test  manager  and  the  facility  managers  is  configuration 
management.  Again,  because  the  facilities  are  shared  by  many  users  with  different  requirements, 
the  program/test  manager  needs  each  facility  to  demonstrate  a  process  to  guarantee  the  facility  is 
configured  properly  for  each  integration  and  test  event.  Often,  facility  managers  will  claim  that 
configuration  management  is  a  normal  business  practice  that  the  program/test  manager  should  just 
accept.  JADS’  experience  with  multiple  facilities  shows  that  every  facility  is  likely  to  have 
problems  returning  to  a  configuration  multiple  times  over  a  period  of  time.  The  cost  to  the 
program/test  manager  may  be  a  wasted  integration  or  test  event,  one  that,  due  to  the  scheduling 
issues  discussed  above,  may  have  significant  impact  on  the  activity. 

4.6  Step  6.  Execute  Distributed  Test  and  Analyze  Results 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  execute  the  distributed  test, 
process  the  output  data  from  the  test  execution,  report  results,  and  archive  reusable  test 
products.  The  key  activities  for  this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure 
7. 
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Figure  7.  Execute  Distributed  Test  and  Analyze  Results 

4.6.1  Activity  6.1  -  Execute  Distributed  Test 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  exercise  all  distributed  test 
participants  as  an  integrated  whole  to  generate  required  outputs  and  thus  achieve  the  stated  test 
objectives.  The  inputs  for  this  activity  are  the  refined  test  procedures  and  tested  ADS 
architecture  from  integration  testing.  The  output  is  the  raw  test  results. 

Again,  one  of  the  key  challenges  for  the  program/test  manager  during  execution  is  related  to 
scheduling.  In  the  case  of  execution,  the  ability  of  the  test  manager  to  manage  without  the  entire 
environment  is  denied.  Also,  depending  on  a  facility  configuration,  the  entire  facility  may  have  to 
be  dedicated  to  the  test.  In  the  JADS  EW  Test  example  above,  the  requirement  for  JADS  was  for 
two  consecutive  weeks  of  testing  to  complete  the  test  matrix.  A  comparison  of  the  ACETEF  and 
Air  Force  Electronic  Warfare  Evaluation  Simulator  (AFEWES)  calendars  for  January  through 
June  1999  yielded  only  two  weeks  where  JADS  could  conduct  its  test.  However,  the  test 
scheduled  immediately  prior  to  the  JADS  test  at  ACETEF  was  the  highest  priority  aircraft 
program  in  the  Navy,  and  if  they  did  not  complete  their  planned  testing  on  time,  the  JADS  test 
would  be  bumped. 

A  related  challenge  for  the  program/test  manager  is  environment  reliability  or  availability.  Test 
facilities  and  laboratories  are  not  built  with  high  reliability  as  a  requirement.  They  do  not  have 
redundant  systems  to  bring  on  line  in  case  of  failures.  Also,  due  to  the  lack  of  resources  to  keep 
up  with  technology  in  these  facilities,  quite  often  the  systems  you  are  using  in  your  environment 
may  be  very  old.  For  example,  in  one  facility' JADS  used,  several  of  the  computers  supporting  the 
environment  were  vintage  1970  and  1980  systems.  Combining  a  lack  of  reliability  and  the 
complexity  of  some  distributed  environments  (one  JADS  test  utilized  approximately  85  different 
computers)  results  in  a  high  potential  for  loss  of  scheduled  test  time.  The  program  manager  must 
allow  for  lost  test  periods  when  scheduling  test  execution. 

4.6.2  Activity  6.2  -  Process  Output 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  post-process  (as  necessary)  the 
output  collected  during  the  test  execution.  The  input  to  this  activity  is  the  raw  test  results  from 
test  execution.  The  output  is  derived  test  results. 

4.6.3  Activity  6.3  -  Prepare  Results 

According  to  the  FEDEP  model,  this  activity  has  two  purposes:  (1)  to  evaluate  the  data  analysis 
results  in  order  to  determine  if  all  test  objectives  have  been  met,  and  (2)  to  identify  legacy 
products  and  make  them  available  to  other  programs.  The  input  to  this  activity  is  the  derived 
test  results,  along  with  the  test  evaluation  criteria  from  Activity  2.3.  The  outputs  are  documented 
test  results  and  legacy  products. 
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5.0  Conclusion 


Planning  and  implementation  of  a  distributed  test  presents  the  program/test  manager  with 
programmatic  as  well  as  technical  challenges.  A  successful  program/test  manager  will  be 
prepared  to  meet  these  challenges  and  produce  and  implement  a  plan  that  will  completely  test  the 
system  and  produce  adequate  insight  into  the  system  performance  and  military  worth  to  justify  a 
production  decision. 
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7.0  Acronyms  and  Definitions 


ACETEF 

Air  Combat  Environment  Test  and  Evaluation  Facility,  Patuxent  River, 
Maryland;  Navy  facility 

ADS 

advanced  distributed  simulation 

AFEWES 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas;  Air 
Force  managed  with  Lockheed  Martin  Corporation 

C4ISR 

command,  control,  communications,  computers,  intelligence,  surveillance  and 
reconnaissance 

COI 

critical  operational  issue 

CONOPS 

concept  of  operations 

DAA 

Designated  Approval  Authority 

DIS 

distributed  interactive  simulation 

DISN 

Defense  Information  Systems  Network 

DITSCAP 

Department  of  Defense  Information  Technology  Security  Certification  and 
Accreditation  Program 

DMAP 

data  management  and  analysis  plan 

DMSO 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

DoD 

Department  of  Defense 

DSI 

Defense  Simulation  Internet 

DT&E 

developmental  test  and  evaluation 

ETE 

JADS  End-to-End  Test 

EW 

electronic  warfare;  JADS  Electronic  Warfare  Test 

FED 

federation 

FEDEP 

federation  development  and  execution  process 

FOM 

federation  object  model 

HLA 

high  level  architecture 

HWIL 

hardware-in-the-loop 

ICD 

interface  control  document 

IPT 

integrated  product  team 

JADS 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

JT&E 

joint  test  and  evaluation 

JTF 

joint  test  force 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

MSRR 

modeling  and  simulation  resource  repository 

O&M 

operation  and  maintenance 

ORD 

operational  requirements  document 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

operational  test  and  evaluation 

OTP 

operational  test  plan 

PID 

program  introduction  document 

RCM 

requirements  correlation  matrix 
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RDT&E 

research,  development,  test  and  evaluation 

ROM 

rough  order  of  magnitude 

SIPRNET 

Secret  Internet  Protocol  Router  Network 

SIT 

JADS  System  Integration  Test;  system  integration  test 

SOC 

statement  of  capability 

SOM 

simulation  object  model 

STAR 

system  threat  assessment  report 

SUT 

system  under  test 

T&E 

test  and  evaluation 

TEMP 

test  and  evaluation  master  plan 

TRC 

test  requirements  council 

W&A 

verification,  validation  and  accreditation 

WBS 

work  breakdown  structure 
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